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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.1 14. Applicant's submission filed on 4 June 2009 has been entered. 

2. Claims 10, 15 and 16 are amended, claims 1 1-14 are as previously presented and claims 
1-9 were previously cancelled. 

Specification 

3. The abstract of the disclosure is objected to because the latest version of the abstract 
received 4 June 2009 is composed of 177 words which still exceeds the 150 word limit imposed 
upon an abstract. Correction is required. See MPEP § 608.01(b). 

4. The amendment of the claims has necessitated the new grounds of rejection that follow. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 
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1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 

5. Claims 10 and 13-16 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Williamson et al. (U. S. Patent Application Publication 2004/0104935 Al, hereafter '935) and 
further in view of Jaszlics et al. (U. S. Patent 6,166,744, already of record, hereafter '744). 

6. In regard to claim 10 (Currently Amended), Williamson teaches an image processing 
apparatus for composting an image of a virtual object and an image of a physical space to 
generate a mixed reality image and causing an HMD to display the mixed reality image ('935; 
abstract; fig. 6; U 0224), comprising: a database which holds data used for generating the image 
of the virtual object ('935; fig. 1; 3D model extraction and hard disk storage; H 0060; H 0069- 
0074); an image capturing unit which is attached to the HMD and captures the image of the 
physical space ('935; HMD with a camera attached, ^ 0224); a first measurement unit which 
measures a position and orientation of the HMD ("935; camera and targets; abstract; ]] 0021- 
0024); but does not teach an object manipulation unit which is used by a user in order to operate 
a position and orientation of the virtual object; a second measurement unit which measures a 
position and orientation of said object manipulation unit; an operation panel which is positioned 
in the physical space, displays an operation panel image used for editing the virtual object, and is 
capable of receiving a user instruction of editing the virtual object; an operation panel image 
generation unit which generates the operation panel image by using the data held in said 
database , and outputs the generated operation panel image to said operation panel ; a rendering 
unit which updates the data held in said database according to the user instruction received via 
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said operation panel and the measurement result of said second measurement unit, and renders, 
by using the updated data, the image of the virtual object according to the measurement results of 
said first and second measurement units; and continues teaching a composition unit which 
composites the image of the virtual object rendered by said rendering unit and the image of the 
physical space captured by said image capturing unit to generate the mixed reality image ('935; 
fig. 17, element 1707; U 0297-0299); and an HMD which displays the mixed reality image 
generated by said composition unit ('935; fig. 17, element 1707; U 0297-0299). Jaszlics, 
working in the same field of endeavor, however teaches an object manipulation unit which is 
used by a user in order to operate a position and orientation of the virtual object ('744; joystick; 
col. 10, In. 35-38; col. 1 1, In. 47-56); a second measurement unit which measures a position and 
orientation of said object manipulation unit; ('744; sensors within the joystick that determine the 
position of the control stick; col. 1 1, In. 47-56) an operation panel which is positioned in the 
physical space ('744; fig. 16, Display A), displays an operation panel image ('744; fig. 21) used 
for editing the virtual object, and is capable of receiving a user instruction of editing the virtual 
object ('744; fig. 21; col. 15, In 5-25) ); an operation panel image generation unit which 
generates the operation panel image by using the data held in said database ('744; fig. 20 & 22; 
virtual simulation computer) , and outputs the generated operation panel image to said operation 
panel ('744; fig. 20 & 22; virtual simulation computer); a rendering unit which updates the data 
held in said database according to the user instruction received via said operation panel and the 
measurement result of said second measurement unit, and renders, by using the updated data, the 
image of the virtual object according to the measurement results of said first and second 
measurement units ('744; fig. 2, 20 & 22; virtual simulation computer; col. 8, In. 6-50) for the 
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benefit of allowing the user to control the virtual object(s) within the mixed reality environment. 
It would have been obvious to one of ordinary skill in the art at the time of the invention to have 
combined the VR teachings of Williamson using a head mounted display coupled with a head 
mounted camera and head/body location techniques with the virtual object controlling teachings 
of Jaszlics for the benefit of allowing the user to control the virtual object(s) within the mixed 
reality environment. 

7. Regarding claim 13 (Previously Presented), Williamson and Jaszlics teach the apparatus 
according to claim 10 and Jaszlics further teaches wherein said operation panel includes a 
display device ('744; fig. 16, Display A) and an operation device ('744; joystick; col. 10, In. 35- 
38; col. 1 1, In. 47-56), wherein the display device displays the operation panel image, and 
wherein the operation device is used for inputting the user instruction. 

8. In regard to claim 14 (Previously Presented), Williamson and Jaszlics teach the apparatus 
according to claim 10 and Williamson further teaches wherein the HMD can present the mixed 
reality image to two eyes of a user who wears the HMD ('935; fig. 16, element 1603; U 0293- 
0295). 

9. Regarding claim 15 (Currently Amended) Williamson teaches an image processing 
method of composting an image of a virtual object and an image of a physical space to generate a 
mixed reality image and causing an HMD to display the mixed reality image ('935; abstract; fig. 
6; Tf 0224), comprising the steps of: holding data used for generating the image of the virtual 
object in a database ('935; fig. 1; 3D model extraction and hard disk storage; If 0060; If 0069- 
0074); providing an image capturing unit, which is attached to the HMD ('935; HMD with a 
camera attached, H 0224), to capture the image of the physical space ('935; abstract; fig. 6; U 
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0224); measuring a position and orientation of the HMD with a first measurement unit ("935; 
camera and targets; abstract; ]f 0021-0024); but does not teach operating an object manipulation 
unit, by a user, in order to position and orient the virtual object; measuring a position and 
orientation of the object manipulation unit with a second measurement unit; positioning an 
operation panel in a physical space to display an operation panel image used for editing the 
virtual object, the operation panel being capable of receiving a user instruction of editing the 
virtual object; generating the operation panel image by using the data held in the database with 
an operation panel image generation unit and outputting the generated operation panel image to 
the operation panel ; updating the data held in the database according to the user instruction 
received via the operation panel and the measurement result of the second measurement unit, and 
rendering, by using the updated data, the image of the virtual object according to the 
measurement results of the first and second measurement units; and Williamson continues 
teaching compositing the rendered image of the virtual object and the captured image of the 
physical space to generate the mixed reality image ('935; fig. 17, element 1707; U 0297-0299); 
and displaying the generated mixed reality image on an HMD ('935; fig. 17, element 1707; Tf 
0297-0299). Jaszlics, working in the same field of endeavor, however teaches operating an 
object manipulation unit, by a user, in order to position and orient the virtual object ('744; 
joystick; col. 10, In. 35-38; col. 1 1, In. 47-56); measuring a position and orientation of the object 
manipulation unit with a second measurement unit ('744; sensors within the joystick that 
determine the position of the control stick; col. 1 1, In. 47-56); positioning an operation panel in a 
physical space to display an operation panel image used for editing the virtual object ('744; fig. 
16, Display A), the operation panel being capable of receiving a user instruction of editing the 
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virtual object ('744; fig. 21; col. 15, In 5-25); generating the operation panel image by using the 
data held in the database with an operation panel image generation unit and outputting the 
generated operation panel image to the operation panel ('744; fig. 20 & 22; virtual simulation 
computer); updating the data held in the database according to the user instruction received via 
the operation panel and the measurement result of the second measurement unit, and rendering, 
by using the updated data, the image of the virtual object according to the measurement results of 
the first and second measurement units ('744; fig. 2, 20 & 22; virtual simulation computer; col. 
8, In. 6-50) for the benefit of allowing the user to control the virtual object(s) within the mixed 
reality environment. It would have been obvious to one of ordinary skill in the art at the time of 
the invention to have combined the VR teachings of Williamson using a head mounted display 
coupled with a head mounted camera and head/body location techniques with the virtual object 
controlling teachings of Jaszlics for the benefit of allowing the user to control the virtual 
object(s) within the mixed reality environment. 

10. 16. (Currently Amended) A computer-readable storage medium encoded with a computer 
program ('935; fig. 16; computer and memory) for an image processing method of composting 
an image of a virtual object and an image of a physical space to generate a mixed reality image 
and causing an HMD to display the mixed reality image ('935; abstract; fig. 6; ]f 0224), 
comprising the steps of: holding data used for generating the image of the virtual object in a 
database ('935; fig. 1; 3D model extraction and hard disk storage; If 0060; | 0069-0074); 
providing an image capturing unit, which is attached to the HMD ('935; HMD with a camera 
attached, If 0224), to capture the image of the physical space; measuring a position and 
orientation of the HMD with a first measurement unit ("935; camera and targets; abstract; ]f 
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0021-0024); but does not teach operating an object manipulation unit, by a user, in order to 
position and orient the virtual object; measuring a position and orientation of the object 
manipulation unit with a second measurement unit; positioning an operation panel in a physical 
space to display an operation panel image used for editing the virtual object, the operation panel 
being capable of receiving a user instruction of editing the virtual object; generating the 
operation panel image by using the data held in the database with an operation panel image 
generation unit and outputting the generated operation panel image to the operation panel ; 
updating the data held in the database according to the user instruction received via the operation 
panel and the measurement result of the second measurement unit, and rendering, by using the 
updated data, the image of the virtual object according to the measurement results of the first and 
second measurement units; and Williamson continues teaching compositing the rendered image 
of the virtual object and the captured image of the physical space to generate the mixed reality 
image ('935; fig. 17, element 1707; ]f 0297-0299); and displaying the generated mixed reality 
image on an HMD ('935; fig. 17, element 1707; ^ 0297-0299). Jaszlics, working in the same 
field of endeavor, however teaches operating an object manipulation unit, by a user, in order to 
position and orient the virtual object ('744; joystick; col. 10, In. 35-38; col. 1 1, In. 47-56); 
measuring a position and orientation of the object manipulation unit with a second measurement 
unit ('744; sensors within the joystick that determine the position of the control stick; col. 1 1, In. 
47-56); positioning an operation panel in a physical space to display an operation panel image 
used for editing the virtual object ('744; fig. 16, Display A), the operation panel being capable of 
receiving a user instruction of editing the virtual object; generating the operation panel image by 
using the data held in the database with an operation panel image generation unit and outputting 
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the generated operation panel image to the operation panel ('744; fig. 20 & 22; virtual simulation 
computer); updating the data held in the database according to the user instruction received via 
the operation panel and the measurement result of the second measurement unit ('744; fig. 2, 20 
& 22; virtual simulation computer; col. 8, In. 6-50), and rendering, by using the updated data, the 
image of the virtual object according to the measurement results of the first and second 
measurement units ('744; fig. 2, 20 & 22; virtual simulation computer; col. 8, In. 6-50) for the 
benefit of allowing the user to control the virtual object(s) within the mixed reality environment. 
It would have been obvious to one of ordinary skill in the art at the time of the invention to have 
combined the VR teachings of Williamson using a head mounted display coupled with a head 
mounted camera and head/body location techniques with the virtual object controlling teachings 
of Jaszlics for the benefit of allowing the user to control the virtual object(s) within the mixed 
reality environment. 

11. Claims 11-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Williamson 
et al. (U. S. Patent Application Publication 2004/0104935 Al, hereafter '935) as applied to 
claims 10 and 13-16 above and in view of Jaszlics et al. (U. S. Patent 6,166,744, already of 
record, hereafter '744) as applied to claims 10 and 13-16 above, and further in view of Lescinsky 
et al. ("Interactive Scene Manipulation in the Virtue3D System," ACM Weg3D'02, pp. 127-135, 
February 24-28, 2002, Tempe, Arizona, ACM 1-581 13-468-1/02/0002, already of record, 
hereafter '468). 

12. Regarding claim 1 1 (Previously Presented), Williamson and Jaszlics teach the apparatus 
according to claim 10 but do not teach wherein the image of the virtual object is generated on the 
basis of 3D CAD data of the virtual object, and said operation panel displays an assembly tree 



Application/Control Number: 10/594,114 Page 10 

Art Unit: 2628 

based on the 3D CAD data. Lescinsky, working in the same field of endeavor, however, teaches 
wherein the image of the virtual object is generated on the basis of 3D CAD data of the virtual 
object, and said operation panel displays an assembly tree based on the 3D CAD data ('468; fig. 
4, pg. 133, § 8-9) for the benefit of providing tools to allow easy to use scene authoring and 
manipulation of content within the virtual 3D mixed reality environment. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to have combined the VR 
apparatus teachings of Williamson and Jaszlics with the explicit VRML tree teachings of 
Lescinsky for benefit of providing tools to allow easy to use scene authoring and manipulation of 
content within the virtual 3D mixed reality environment. 

13. In regard to claim 12 (Previously Presented), Lescinsky further teaches wherein a part, 
which is obtained by enlarging a designated part of the assembly tree included in the operation 
panel image, includes a component name contained in the assembly tree ('468; fig. 4, pg. 133, § 
8-9). 

Response to Arguments 

14. Applicant's arguments with respect to claims 10-16 have been considered but are moot in 
view of the new ground(s) of rejection. 

Conclusion 

The following prior art, made of record, was not relied upon but is considered pertinent to 
applicant's disclosure: 

US 6629065 Bl Methods and apparatus for rapid computer-aided design of objects 

in virtual reality and other environments 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Edward Martello whose telephone number is (571) 270-1883. 
The examiner can normally be reached on M-F 7:30-5:00 EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Xiao Wu can be reached on (571) 272-7761 . The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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Examiner, Art Unit 2628 
/XIAO M. WU/ 

Supervisory Patent Examiner, Art Unit 2628 



